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What You Need 


An OS/2 C compiler and Toolkit. 

- Exercises will include directions for compiling with IBM 
C Compiler 3.x and GCC 3.3.5. 

- GCC 3.3.5 can be provided if necessary. 

At least a basic knowledge of C programming 
(recommended) or REXX. 

- REXX support is somewhat limited; the extent of 
coverage will depend on class interest. 








What is Unicode? 


Unicode is a character standard designed to 
allow all possible characters from all known 
writing systems to be represented on computer 
systems in an interchangeable form. 

- Defines implementation requirements such as 
encoding formats, bidirectional algorithms, rendering 
of composite characters, etc. 

- Assigns every character a unique numeric value 
between 0 and 1,114,111. This assignment is known 
as the Universal Character Set (UCS). 

The terms "UCS" and "Unicode" are usually used 
interchangeably. 








Versions of Unicode 


The Unicode standard is maintained by the 
Unicode Consortium, and is currently up to 
version 5.0 (released July 2006). 

- Version 1.0 was released in 1991 and supported 
65,536 character codepoints (28,302 of which were 
assigned). 

- Starting with version 2.0, the codespace was 
expanded to 17 planes, each supporting 65,536 
codepoints, for a total codespace of 1,114,112 
possible characters (but the additional planes were not 
actually used until version 3.1). 

The Universal Character Set is also published on 
its own by the International Organization for 
Standardization as ISO/IEC 10646. 








Why Unicode? 


Encoding characters by codepage has many 
disadvantages: 

- Each codepage only supports one or two character 
sets (usually basic Latin and up to one other) - difficult 
to support multiple languages simultaneously. 

- Different computer platforms tend to use different 
codepages, even to represent the same character sets 
(e.g. IBM 850 vs Windows 1252) - complicates data 
interchange. 

- DBCS codepages tend to use relatively crude variable- 
width encoding that makes string parsing difficult. 

- Too many codepages, not enough standardization! 








Why Unicode (cont.)? 


Why use Unicode in your application? 

- Access to a wide range of text-manipulation functions 
which are independent of language or encoding. 

- Multiple character sets can be supported 
simultaneously, even in a single output stream. 

- Text can be converted to almost any encoding on 
demand. 








UCS Character Values 


Unicode (UCS) character values (or codepoints) 
are traditionally written as "U+*", where * is the 
character's value in hexadecimal. 

Values U+0000 (0) through U+007F (127) 
correspond to the standard ASCII character set. 

It is therefore easy to convert these characters 
to and from Unicode. 

Values U+0080 (128) through U+00FF (255) 
correspond to the ISO Latin-1 extensions. This 
means that the UCS character mapping of 
U+0000 through U+00FF corresponds exactly to 
ISO-8859-1 (codepage 819). 








The UCS Codespace 


The Universal Character Set encompasses 17 
planes, each containing 65,536 codepoints: 

- Plane 0 (U+0000 - U+FFFF): Basic Multilingual Plane 
(BMP) 

- Plane 1 (U+10000 - U+1FFFF): Supplementary 
Multilingual Plane (SMP) 

- Plane 2 (U+20000 - U+2FFFF): Supplementary 
Ideographic Plane (SIP) 

- Plane 14 (U+E0000 - U+EFFFF): Supplementary 
Special-Purpose Plane (SSP) 

- Planes 15 & 16 (U+F0000 - U + 10FFFF): Supplementary 
Private Use Area 

The BMP contains most characters in use around 
the world today. 








Unicode Character Encoding 


Problem: how can a computer represent each 
character, consistently and unambiguously, as 
an integer value between 0 and 1,114,111? 

Traditionally, 1 byte = 1 character. But this only 
allows 256 values - far too small for the UCS 
codespace! 

Simplest solution: increase the number of bytes 
per character. 

- First implemented in UCS-2 encoding. 

- Later expanded to UCS-4 encoding. 








UCS-2 Encoding 


UCS-2 is the original Unicode encoding. 
Introduced back in the days when UCS only 
defined 65,536 characters (Plane 0). 

- Fixed-width encoding: 2 bytes per character 
(supporting the value range 0 - 65,535). 

- Advantages: simple, makes data processing easy. 

- Disadvantages: not backwards-compatible with ASCII; 
only supports Plane 0. 

- Officially considered obsolete, but still used by some 
implementations. 

(The OS/2 Unicode APIs use UCS-2 internally.) 








UCS-4 Encoding 


UCS-4 was introduced to replace UCS-2 when 

Unicode was expanded to 1,114,112 characters. 

- Fixed-width encoding: 4 bytes per character, 
supporting the entire range of UCS values (and then 
some). 

- Advantages: simple, makes data processing easy; 
supports entire UCS codespace. 

- Disadvantages: requires a huge amount of storage; 
not backwards-compatible with ASCII, or even with 
UCS-2. 

- Rarely used. 

Nowadays, UCS-4 is more commonly called 

"UTF-32". 








Problems with Fixed-Width 

Encoding 

Once the UCS codespace was expanded to 17 
planes, fixed-width character encoding became 
impractical. 

- Four bytes per character is wasteful, especially since 
(in practice) most characters fall within Plane 0 (BMP). 
(See example.) 

- Most Unicode libraries were originally designed to use 
two-byte characters (UCS-2); redesigning them for 
four-byte characters (UCS-4) can break compatibility. 

- Fixed-width multi-byte encodings are not compatible 
with ASCII data streams (which are widely used). 

- UCS-4 isn't even compatible with UCS-2 data streams! 

A better solution: variable-width encoding. 








UTF-16 Encoding 


Intended to seamlessly replace UCS-2. 

- Uses 2 bytes per character to encode characters 
within the BMP (U+0000 - U+FFFF). 

- Uses 4 bytes per character to encode characters 
outside the BMP (U+1FFFF and up). Leading bytes of 
4-byte characters always fall within the range U+D800 
- U+DFFF, which are reserved within the BMP and so 
cannot start legal 2-byte characters. 

- Any legal UCS-2 character is also a legal UTF-16 
character. 

- Advantages: backwards-compatible with UCS-2; 
requires no more than 2 bytes to represent any BMP 
character; supports the entire UCS codespace. 

- Disadvantages: slightly more complex than UCS-2; still 
not backwards-compatible with ASCII data streams. 








UTF-8 Encoding 


Possibly the most ingenious encoding of all. 

- Uses between 1 and 4 bytes for each character: 

• 1 byte for values U+0000 - U+007F (basic ASCII) 

• 2 bytes for values U+0080 - U+07FF 

• 3 bytes for values U+0800 - U+FFFF 

• 4 bytes for values U+10000 and up. 

- A 1-byte character always starts with a leading 0 bit. 

- The first byte of a multi-byte character always starts 
with a number of leading 1 bits equal to the number of 
bytes used (i.e. llOxxxxx, lllOxxxx or llllOxxx). 
Following bytes always start with 10. The remaining 
bits are used to encode the character value. 

- All ASCII text (i.e. characters below U+0080) is also 
valid UTF-8 text! 








UTF-8 Encoding (cont.) 


Advantages of UTF-8: 

• Backwards-compatible with basic ASCII. 

• Supports the entire UCS codespace. 

• Requires much less storage than UCS-4, and (on average) 
slightly less than UCS-2. 

• Every single byte in a data stream is instantly identifiable as a 
leading byte, a following byte, or a single-byte character 
(which helps prevent data corruption). 

Disadvantages: 

• Relatively complex algorithm, resulting in higher processing 
overhead. 

UTF-8 is typically used for text output and interchange, 
not for internal processing. 








Unicode Under OS/2 


The OS/2 implementation of Unicode uses UCS-2 
for internal processing. 

Characters outside the BMP are not supported 
(AFAIK). 

OS/2 supports three Unicode encodings: UCS-2, 
UTF-8, and UPF-8 (an OS/2-specific proprietary 
implementation used for output under PM). 

The OS/2 Unicode API (Universal Language 
Support) is available under Warp 4 and up, or 
Warp 3 with a recent FixPak. (Some versions of 
the Java 1.1.8 runtime package will also install or 
update it as needed.) 








The Unicode API 


The OS/2 Unicode functions (a.k.a. "Universal 
Language Support") cover four major categories 

- Unicode Text Processing: handling Unicode (UCS-2) 
text. 

- Conversion : converting text to and from Unicode, or 
from one codepage to another. 

- Localization : presenting information according to 
national or cultural conventions. 

- Keyboard Input: converting keyboard input to and 
from Unicode according to different keyboard layouts. 








Using the Unicode API 


Header files: 

- uni def . h: main API definitions (text processing & 
localization) 

- uconv. h: conversion API definitions 

- uni kbd. h: keyboard API definitions 

Import libraries (when using the IBM Toolkit): 

- l i bui s . l i b: text processing & localization functions 

- l i bconv. l i b: conversion functions 
-uni kbd. lib: keyboard functions 








Representing Unicode Text: 

UniChar 

All Unicode text is encoded internally as UCS-2. 

A Unicode character is represented by the 
UniChar data type (a two-byte integer value): 


A UniChar is also sometimes referred to as a 
"UCS code element". 








Setting UniChar Values 


Directly assign the character's UCS value as an 
integer. 
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Use C wide-character conventions (assuming 
wchar_t is an unsigned short): 
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Only use this technique when dealing with basic 
ASCII characters! 








Setting UniChar Values (cont.) 


Use the codepage conversion functions: 

- UniStrToUcsQ and UniUconvToUcsQ will convert a 
multibyte string from any known codepage into a 
UniChar string. 

- UniStrFromUcsO and UniUconvFromUcsO will convert a 
UniChar string into a multibyte string in the desired 
codepage. 








Working with UniChar Strings 


• Take the string: 

/ / o Wc r f <£ 

• Under a normal (ASCII-based) multibyte 
codepage, this string is represented by: 

f 6 6 5 6 % % 6 1? ;Z o J J 6 2^ J X 6 TJ 6 f oo 

• As a UniChar string, it becomes: 

00 48 00 65 00 6C 00 6C 00 6F 00 20 00 77 00 6F 00 72 00 6C 
00 64 00 00 








Working with UniChar Strings 

(cont.) 

Most standard string-handling functions treat a 
zero byte as a string termination character, and 
therefore will not handle this string correctly. 

As mentioned, C wide-character APIs can 
sometimes be used (where available). But: 

- In practice, this is only possible with basic ASCII text 
(since these APIs cannot convert the Unicode values 
into meaningful characters for the current codepage). 

- Also, there is no way to use wide-character 
conventions in conjunction with (for instance) PM 
controls. 

Consequently, ULS provides its own functions for 
handling UniChar text. 








Unicode Input and Output 


In general, UniChar strings must be converted to 
another format before output. 

- Convert to a Unicode encoding which is designed for 
output: 

• UPF-8 (codepage 1207) - preferred for use under Presentation 
Manager. 

• UTF-8 (codepage 1208) - preferred for data interchange (files, 
e-mail, etc.). 

This does not allow direct display in OS/2 text sessions. 

- Convert to another codepage. This allows display of 
text in OS/2 window or full-screen sessions by 
converting to the current process codepage. 

Conversely, input routines can accept text in the 
current codepage, and then convert it to UCS-2 
for subsequent processing. 








Unicode String Manipulation 


Most standard library functions cannot be used 
with UniChar text (due to embedded zero bytes). 
Therefore, ULS includes Unicode-enabled 
equivalents of several standard C library 
functions: 

- The str*() functions of the standard string library. 

- The character transformation functions tolower() and 
toupper(). 

In addition, several functions are provided for 
querying and transforming UniChar characters 
and strings in a locale-dependent way. 








Exercise 1: 

Using UniChar Strings 

Exercise: 

- Review the sample program. Make sure you 
understand what it does. 

- Verify that the program compiles and runs 
successfully. 

Objectives: 

- Verify compiler and toolkit setup. 

- Verify program using ULS API can be compiled and 
run. 

- A brief introduction to UniChar string manipulation. 








Codepage Conversion 


The ULS conversion functions allow text to be 
converted from any multibyte codepage into the 
equivalent strings in UCS-2 (UniChar) format, 
and vice versa. 

Text can also be converted from one multibyte 
codepage to another by going through UCS-2 as 
an intermediate step. 

Logical character value (not byte value) is 
preserved across codepages, whenever possible. 

If a character does not exist in the target 
codepage, it is replaced by a designated 
"substitution character". 








UconvObject 


The UconvObject type is used to control 
conversions (in either direction) between UCS-2 
and one particular codepage. 

Use UniCreateUconvObjectO to create a 
UconvObject for the specified codepage: 

UconvObject uconv; 

UniCreateUconvObject( (UniChar*) L"IBM-850" / &uconv ); 


Use UniFreeUconvObjectO when the 
UconvObject is no longer required: 


UniFreeUconvObject( uconv ); 








Conversion Specifiers 


The first parameter to UniCreateUconvObjectO is 
a UniChar string called the conversion specifier: 

" < codepage-name > [ @ ] [ < modifers > ]" 

The codepage name must be a legal OS/2 
codepage identifer, normally in the form "IBM-x" 
where x is the codepage number. Certain 
aliases are also defined (in 
\ TlZ&lEL 1350, and some 

constants are provided in uccnv. /?. 

UniMapCpToUcsCpO will generate a legal 
codepage name from the specified codepage 
number. 








Conversion Modifiers 


The optional modifers define certain attributes of 
the conversion object (separated by commas). 

- map: Defines how control bytes should be treated 
during conversion. Default: "map=data" 

- path: Indicates whether strings are assumed to 
contain paths (DBCS only). Default: “path=yes” 

- endian: Indicates the UCS-2 byte order (endian) to 
use. Default: “endian=system" 

- sub: Indicates whether character substitution is 
enabled. Default: ", sub=from-ucs" 

- subchar: Indicates the substitution character to use 
in codepage strings. (Default varies by codepage.) 

- subuni: Indicates the substitution character to use in 
UCS-2 strings. Default: “subuni=\xFFFD" 








Performing Conversions 


Once you have a UconvObject, you can two 
different sets of functions to perform the 
conversion(s). 

- UniStrToUcsO and UniStrFromUcsQ 

- UniUconvToUcsQ and UniUconvFromUcsO 

The UniStr*() functions are easier to use; 
however, the UniUconv*() functions allow for 
slightly more flexible error-checking and 
recovery. 

The UniStr*() functions were not available in the 
first versions of the Unicode API; they were 
added in an early Warp 4 FixPak (and a late 
Warp 3 FixPak). 








UniStrTollcs / UniStrFromUcs 


UniStrToUcsO and UniStrFromUcsO use a fairly 
simple syntax. 

Conversion is atomic: it either succeeds or it fails 
(reflected in the return code). 

Substitution is always performed (regardless of 
the conversion modifers used to create the 
UconvObject). 

Parameters: 

- UconvObject 

- Output buffer (must be already allocated) 

- Input string 

- Length of output buffer (in characters) 








UniUconvToUcs / 
UnillconvFromUcs 


UniUconvToUcsO and UniUconvFromUcsO use a 
fairly complex syntax. 

An error may cause conversion to stop part-way 
through, and more information is available to 
allow error recovery. 

Parameters: 

- UconvObject 

- Pointer to input string (must already be allocated) 

- Pointer to input string length (in characters) 

- Pointer to output buffer (must already be allocated) 

- Pointer to output buffer length (in characters) 

- Pointer to number of substitutions made 








Output Buffer Length 


The output buffer must be large enough to 
contain all converted characters, plus NULL. 

- When converting to UCS-2, the UniChar output buffer 
will never be longer (in UniChars) than the length of 
the input buffer (in bytes), not including the 
terminating NULL. 

- When converting from UCS-2, the output string may 
be longer than the UniChar input string, if the target 
codepage allows 2-, 3-, or 4-byte characters. 

• If you know the target codepage contains only single-byte 
characters, the output buffer length may be the same as the 
input length (not including the terminating NULL). 

• If the character width of the target codepage is undetermined, 
you should allocate at least 4 output bytes per input UniChar, 
because OS/2 MBCS codepages may represent a single 
character using up to 4 bytes. 








Character Substitution 


If the input string contains characters that do 
not exist under the target codepage, the 
character is (normally) replaced in the output 
string by a generic "substitution character". 

Every codepage has its own default substitution 
character (which may be changed through the 
UconvObject attributes). It should always be a 
displayable glyph under the target codepage. 

If substitution is disabled (through the 
UconvObject attributes), an error condition will 
be returned whenever an unsupported character 
is encountered. (Applies to the UniUconv * 
functions only.) 








Exercise 2: 

Codepage Conversion 

Exercise: 

- Review the example program(s). 

- Modify the example program to convert the input text 
to UTF-8 instead of HTML. 

Objectives: 

- Demonstrate the conversion process. 

- Illustrate the difference between the UniStr* and 
UniUconv* functions. 








Localization 


Localization is based on the concept of a locale-. 
a set of conventions associated with a particular 
language or culture that specifies how 
information should be presented. 

These conventions include such things as: 

- The names of the country and language normally 
associated with the locale. 

- The default currency unit. 

- The standard number format. 

- The standard time and date format. 

- The standard units of measurement. 

- The default codepage(s) associated with the locale. 

- Rules for text classification and transformation. 








How Locales Work 


Locales are used for: 

- Determining character types or text transformation 
rules when using certain Unicode functions. 

- Modifying application behaviour as appropriate for the 
current environment. 

There are two types of locale: 

- System locales : standard locale constants defined by 
OS/2. They cannot be modified or deleted. System 
locales exist for every country and language known to 
OS/2, and are used to identify standardized 
conventions. 

- User locales: customizable locale instances which may 
be created, modified, and/or deleted. User locales are 
derived from system locales, but represent the user's 
own preferences rather than generic standards. 








Identifying Locales 


A LocaleObject is used as a handle to a specific 
locale. A variable of this type is passed to 
certain functions in order to identify the locale 
being used. 

UniCreateLocaleObjectO is used to initialize a 
LocaleObject for the specified locale. 

- The locale is usually identified by UCS-2 name 
("en_US", "de_DE_EURO”, etc.). 

- Specifying an empty string will create a locale based 
on the current environment settings (e.g %LANG%). 

A LocaleObject should be freed using 
UniFreeLocaleObjectO once it is no longer 
needed. 








Identifying Locales (cont.) 


UniQueryLocaleListQ can be used to query the 
names of all existing locales (system, user, or 
both). 

UniMapCtryToLocaleO returns the locale name 
corresponding to the specified country code. 

UniQueryLocaleObjectQ can be used to obtain 
the locale name associated with an existing 
LocaleObject. 








Getting Locale Information 


Every locale consists of a fixed number of key- 
value pairs, known as locale items, which 
describe the conventions for that locale. 

Each locale item is referred to by a key constant. 
The prefix indicates the data type: 

- LOCI_s, LOCI j, and LOCI_w indicate string values. 

- LOCI_i and LOCI_x indicate integer values. 

To query the value of a specified locale item: 

- UniQueryLocaleltemQ writes the current value of any 
locale item to a string variable. 

- UniQueryLocaleValueQ writes the current value of an 
integer locale item to an integer variable. 








The Locale Conventions 

Structure 


The locale conventions structure (struct 
UniLconv) contains information about how 
numbers and currency values are represented 
by a locale. 

It is designed to be analogous to the C Iconv 
structure (returned by the localeconv() library 
function). 

This structure may be obtained for a specified 
locale using UniQueryLocalelnfoO. 








Working With User Locales 


User locales are used to represent locally- 
configured preferences. 

Normally, user locales are created and modified 
by the user through the OS/2 Locale applet. 
However, it can also be done programmatically. 

- UniMakeUserLocaleQ creates a new user locale, by 
copying an existing locale (system or user). 

- UniDeleteUserLocaleO deletes a user locale. 

- UniSetUserLocaleltemO modifies individual items 
within a user locale. 

Changes do not take effect outside the current 
process until UniCompleteUserLocaleQ is called. 
This function writes all user locales to disk. 








Exercise 3: 

Getting Locale Information 

Exercise: 

- Review the example program. 

- Modify this program to display selected items from 
each locale (e.g. the associated country and language 
names). 

Objective: 

- Demonstrate how to identify and query locales. 








Character Attributes 


A character attribute (or classification) describes 
categories to which a character belongs. 

Attributes include standard POSIX types like 
"alpha" and "digit", but also extended Unicode 
types like "ideograph" and "nonspacing". 

There are also attributes for describing text 
layout, and others indicating the specific 
Unicode character sets to which a character 
belongs. 








Attribute Names and Identifiers 


• Every attribute has a integer identifier 
(referenced by symbolic constant). 

• Most (though not all) attributes also have a 
name, which is a human-readable UniChar 
string. 

• UniQueryAttrO may be used to obtain the 
identifier value associated with an attribute 
name. 








Attribute Categories 


Character attributes are grouped into several 
different categories: 

- Character type attributes (names start with a 
lowercase letter): 

• POSIX types (identifier symbols start with CT_) 

• Extended types (identifier symbols start with C3_) 

• Win32 compatibility types (identifier symbols start with Cl 
names) 

- Character set attributes (names start with 

- Layout/BIDI attributes (names start with “#”) 








Localized Attributes 


An AttrObject is used to represent attributes in 
a locale-specific context. 

- UniCreateAttrObjectO creates an AttrObject for the 
specified attribute(s) and locale. 

- UniFreeAttrObjectQ frees an AttrObject when it is no 
longer needed. 

- UniQueryCharAttrQ queries a UCS-2 character's 
attributes using an AttrObject. 

- UniScanForAttrO searches a UniChar string for 
characters matching the specified AttrObject. 

- Standard POSIX attributes may be queried using 
various UniQuery* functions (similar to the C "ctype" 
functions). 








Locale-Independent Attribute 

Functions 


There are several functions which may be used 
query attributes independently of locale. 

- UniQueryCharQ is used to determine whether a single 
UCS-2 character has the specified POSIX or Win32 
character-type attributes (CT_* or Cl *) 

- UniQueryStringTypeO returns an array of integer 
bitmasks for a UniChar string, where each bitmask 
describes the attributes (in the requested category) of 
a single character. 

- UniQueryCharTypeO returns all the attributes of a 
single UCS-2 character in a UNICTYPE data structure. 








Transforming Text 


ULS provides various functions for transforming 
UniChar strings in locale-specific ways. 

- UniTransLowerO converts text to lowercase. 

- UniTransUpperO converts text to uppercase. 

- UniTransformStrO performs advanced string 
transformations. 

Note that the input and output strings are not 
guaranteed to be the same length. Make sure 
you provide a large enough buffer! 








XformObjects 


An XformObject is used to define a 
transformation, for use with UniTransformStr(). 
It has two attributes: 

- The transformation type 

- The locale being used 

Use UniCreateTransformObjectO to create an 
XformObject. 

Use UniFreeTransformObjectO to dispose of it 
when done. 








Transformation Types 


The following transformation types are valid for 
all locales: 

- lower: Convert text to lowercase. 

- upper: Convert text to uppercase. 

- compose: Convert character-diacritic combinations 
into single glyph forms. 

- decompose: Convert diacritical forms into separate 
characters and diacritics (combining marks). 

- hiragana: Convert Japanese phonetic characters into 
Hiragana. 

- katakana: Convert Japanese phonetic characters into 
Katakana. 

- kana: Convert Japanese phonetic characters into half¬ 
width Katakana. 








Exercise 4: 
Transforming Text 

Exercise: 

- Review the example program. 

- Modify this program to remove all accents from the 
string (i.e. convert accented characters into non- 
accented ASCII characters). 

Objective: 

- Demonstrate Unicode text transformations. 








Additional Information 


OS/2 Toolkit documentation is obsolete and/or 
incomplete. Project to provide updated 
documentation: 

http://www.cs-club.org/~alex/os2/toolkits/uls/index.html 

General Unicode information is available from 
the Unicode Consortium: 

http://www.unicode.org/ 

Useful information & samples from one of the 
OS/2 Internationalization developers: 

http://www.borgendale.com/uls.htm 








Questions? 









